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(54) Scheme for multimedia data retrieval using event names and time or location information 



(57) In a multimedia data retrieval device, each one 
of a plurality of multimedia data is managed in relation 
to time information and/or location information indicating 
a time and/or a location at which each multimedia data 
is created. Then, the time information and/or the location 



information corresponding to a retrieval request are ob- 
tained upon receiving the retrieval request specified by 
using event names, and multimedia data are retrieved 
from the plurality of multimedia data according to the ob- 
tained time information and/or location information. 
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Description 

BACKGROUND OF THE INVENTION 
FIELD OF THE INVENTION 

[0001] The present invention relates to a scheme for 
retrieving a desired multimedia data from a plurality of 
multimedia data. 

DESCRIPTION OF THE BACKGROUND ART 

[0002] Due to the advance of instruments such as dig- 
ital cameras (digital still cameras), image scanners, and 
file scanners, digital video cameras, it has become pos- 
sible to easily produce multimedia data containing dig- 
ital still images (photographs, documents, etc.), digital 
moving pictures (video, etc.) and digital audio data or 
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lized in a variety of fields. 

[0003] The multimedia data are often managed by us- 
ing a file system or database management system of a 
computer or a dedicated software, and there have been 
many propositions for making a retrieval of a necessary 
multimedia data easier. 

[0004] Conventionally, the widely used multimedia 
data management method has been a method for clas- 
sifying a plurality of files containing multimedia data 
such as still images, moving pictures, audio data, etc., 
by creating groups called directories or folders. 
[0005] Fig. 33 shows a concrete example of this meth- 
od. In the example shown in Fig. 33, two directories 
"1 970" and "1 998" are created below a directory "PHO- 
TO", and two directories "ISE-SHIMA" and "JAPAN EX- 
PO" are created below the directory "1970", and two di- 
rectories "KYOTO" and "NAGANO OLYM PICS" are cre- 
ated below the director "1998". 
[0006] In addition, below the directory "ISE-SHIMA", 
"MEOTOIWA-ROCKS.JPG" which is data for a photo- 
graph of the "Meotoiwa rocks" and "ISE-JINGU- 
SHRINE.JPG" which is data for a photograph taken at 
the "Ise-jingu shrine", both of which are taken by a digital 
camera during a trip to Ise-shima are managed. 
[0007] Similarly, below the directory "JAPAN EXPO", 
"TOWER-OF-THE-SUN.JPG" which is data for a pho- 
tograph taken at the 'Tower of the sun" and "U.S.-PA- 
VILION.JPG" which is data for a photograph taken at 
the "U.S. pavilion", both of which are taken at the EXPO 
site during a trip to Osaka to see the Japan EXPO, as 
well as "OSAKA-JO-CASTLE.JPG" which is data for a 
photograph taken at the "Osaka-jo castle" on the way 
back from that trip are managed. 
[0008] Similarly, below the directory "KYOTO". 
"KINKAKUJl-TEMPLE.JPG" which is data for a photo- 
graph of the "Kinkakuji temple" and "K I YO Ml ZUD ERA- 
TEMPLE. JPG" which is data for a photograph of the "Ki- 
yomizudera temple", both of which are taken by a digital 
camera during a trip to Kyoto are managed. 



[0009] Similarly, below the directory "NAGANO OL- 
YMPICS", "JUMP.JPG" which is data for a photograph 
taken at the jump game site and "SPEED-SKATE.JPG" 
which is data for a photograph taken at the speed-skate 

5 game site, both of which are taken during a trip to see 
the Nagano Olympics, as well as "ZENKOJI-TEMPLE. 
JPG" which is data for a photograph taken at the "Zenko- 
ji temple" on the way back from that trip are managed. 
[0010] These photograph data are assumed to be still 

10 image data in the JPEG format in forms of files having 
an extension "JPG", but they can be given in any other 
data format. 

[0011] Now, the conventionally widely used method 
for retrieving such multimedia data has been a method 
15 for attaching keywords for the purpose of retrieval to 
each multimedia data and retrieving each multimedia 
data by using its keyword. 

[0012] Fig. 34 and Fig. 35 show a concrete example 
of this method, wIwr Fig. 35shDws an exemplary con- 
20 tent of the keyword table file "KEYWORD" appearing in 
Fig. 34. 

[0013] In the example shown in Fig. 34 and Fig. 35, 
files for ten photograph data from "P0000100.JPG" to 
"P0000581 .JPG" are managed under a directory "PHO- 

25 TO" in the file system, and a list of keywords attached 
to these individual files are managed in a file "KEY- 
WORD" as a keyword table. Here, the data are man- 
aged by attaching arbitrary number of keywords to each 
data such that B P0000100.JPG" has three keywords 

30 "Meotoiwa rocks", "Ise-shima" and "New year's day", 
"P0000191 JPG" has three keywords "ise-jingu shrine", 
"Ise-shima" and "New year's day", and so on. 
[0014] By managing data in this way, when data for 
photographs taken at Ise-shima are necessary, 

35 "P00001 00.JPG" and "P00001 01 .JPG" which have the 
keyword "Ise-shima" can be selected, for example. The 
keywords can be managed in various methods such as 
a method for creating a file for maintaining correspond- 
ences between file names of photograph data and key- 

40 words attached to them as in the example shown in Fig . 
34 and Fig. 35, or a method using a database manage- 
ment system. 

[0015] The conventional method described above 
can be used for management of data in a variety of for- 
45 mats such as still Images like photograph data used in 
the above example, moving picture data like videos and 
audio data, usual text file like data, some measurement 
data, and links to other data such as URLs of the WEB. 
[0016] For arrangement of photograph data of a dig- 
so ital camera, there is also a method using an album soft- 
ware which arranges photographs on an electronic doc- 
ument in a manner similar to an ordinary album for ar- 
ranging ordinary photographs printed on photographic 
papers. 

55 [0017] In the data formats for photograph data taken 
by a digital camera (or a digital still camera) and video 
data taken by a digital video camera, the attribute infor- 
mation such as a time and a location of the image taking 
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can be recorded, as disclosed in the "digital still camera 
image file format specification (Exit) Version 2.1" (JEI- 
DA-49-1998) by Japan Electronics Industry Develop- 
ment Association, and "Recording-Helical -scan digital 
video cassette recording system using 6.35 mm mag- 
netic tape for consumer use (525-60, 625-50, 1125-60 
and 1250-50 systems)" by 1EC (International Electro- 
technical Commission). 

[001 8] There are some digital cameras and digital vid- 
eo cameras which have a function for automatically re- 
cording the image taking time and location information 
as attributes of data according to these formats, using 
information obtained from a built-in clock and a GPS or 
the like. 

[0019] By utilizing the attribute information such as 
image taking time and image taking location that is au- 
tomatically attached, it is possible to retrieve data by 
specifying the image taking time or longitude and lati- 
tude of the image taking location, without requiring tedi- 
ous tasks of classifying data or attaching keywords as 
in the conventional methods described above. 
[0020] However, in order to retrieve data by specifying 
a location by a set of longitude and latitude, it is neces- 
sary to know the longitude and the latitude of that loca- 
tion in advance. In order to make this retrieval easier, 
there is a scheme for displaying a map on a screen and 
arranging photographs at their image taking locations 
on the map. 

[0021] There is also a scheme for providing a table of 
correspondences between a location name and longi- 
tude and latitude set of a representative point indicated 
by the location name such that when the location name 
is specified, the longitude and latitude set of the corre- 
sponding representative point is checked and photo- 
graphs taken in a vicinity of that point are retrieved. 
[0022] As described, the conventionally widely used 
scheme for managing multimedia data by classifying 
them into directories has a drawback that it requires te- 
dious tasks of classifying data into directories with 
names that are easily comprehensible to the human be- 
ing, and giving easily comprehensible names to individ- 
ual files if necessary. Moreover, when there are data that 
can be classified into more than one groups conceptu- 
ally, it has been necessary to make copies and enter 
them into a plurality of directories or make it appear as 
if one data is virtually existing in a plurality of directories 
by using a link mechanism. 

[0023] Also, the conventionally widely used scheme 
for managing and retrieving multimedia data by attach- 
ing keywords to them has a drawback that it requires 
tedious tasks of attaching keywords. In addition, there 
is also a drawback that keywords that are convenient 
for the retrieval purpose must be attached in order to 
facilitate the data retrieval. 

[0024] Also, the scheme for creating an electronic al- 
bum by using an album software for a digital camera has 
a drawback that the operation to create an album itself 
is tedious. 



[0025] On the other hand, the scheme for utilizing the 
image taking time or image taking location information 
that is automatically attached by a digital camera or a 
digital video camera requires no manual task for attach- 

5 ing keywords or classifying data for the purpose of the 
retrieval. However, in order to carry out the retrieval by 
specifying the time or the location accurately, there is a 
need to know the image taking time or the image taking 
location of the retrieval target more or less accurately in 

10 advance, so that it is difficult to realize a practically us- 
able retrieval. 

[0026] There is also a scheme for providing a table of 
correspondences between a location name and longi- 
tude and latitude set of a representative point indicated 

is by the location name and retrieving data of a nearby lo- 
cation by specifying a location name rather than specif ic 
numerical values of longitude and latitude, but even in 
this scheme, it has been impossible to carry out the re- 
trieval by specifying a concept other than the location 

20 name. 

[0027] In the scheme using directory names or key- 
words, terms that are not location names but represent- 
ing concepts that are easily comprehensible to the hu- 
man being such as "Nagano Olympics" or "Japan EX- 

25 po", for example, can be used for directory names or 
keywords , so that it is possible to realize the retrieval 
that is more intuitive and easily comprehensible. How- 
ever, it has been difficult to realize the retrieval by using 
concepts that are easily comprehensible to the human 

30 being based on the automatically attached information 
such as time and location information alone. 

SUMMARY OF THE INVENTION 

35 [0028] It is therefore an object of the present invention 
to provide a multimedia data retrieval scheme capable 
of realizing flexible retrieval of multimedia data by using 
concepts that are easily comprehensible to the human 
being based on the automatically attached information 

40 such as time and location information, without requiring 
tedious tasks such as classifying multimedia data or at- 
taching keywords for the purpose of the retrieval. 
[0029] It is another object of the present invention to 
provide a multimedia data retrieval scheme capable of 

45 displaying information representing contents recorded 
in multimedia data by using concepts that are easily 
comprehensible to the human being based on the auto- 
matically attached information such as time and location 
information, without requiring tedious tasks such as 

50 classifying multimedia data or attaching keywords in ad- 
vance. 

[0030] According to one aspect of the present inven- 
tion there is provided a multimedia data retrieval meth- 
od, comprising: (a) managing each one of a plurality of 
55 multimedia data in relation to time information and/or lo- 
cation information indicating a time and/or a location at 
which each multimedia data is created; (b) obtaining the 
time information and/or the location information corre- 
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sponding to a retrieval request upon receiving the re- 
trieval request specified by using event names; and (c) 
retrieving multimedia data from the plurality of multime- 
dia data managed by the step (a), according to the time 
information and/or the location information obtained by 5 
the step (b). 

[0031 ] According to another aspect of the present in- 
vention there is provided a multimedia data retrieval de- 
vinft rinmnrifiinn: a data management unit confiaured to 
manage each one of a plurality of multimedia data in to 
relation to time information and/or location information 
indicating a time and/or a location at which each multi- 
media data is created; a processing unit configured to 
obtain the time information and/or the location informa- 
tion corresponding to a retrieval request upon receiving '5 
the retrieval request specified by using event names; 
and a data selection unit configured to retrieve multime- 
dia data from the plurality of multimedia data managed 
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formation and/or the location information obtained by 20 
the processing unit. 

[0032] According to another aspect of the present in- 
vention there is provided a computer usable medium 
having computer readable program codes embodied 
therein for causing a computer to function as a multime- 25 
dia data retrieval device, the computer readable pro- 
gram codes include: a first computer readable program 
code for causing said computer to manage each one of 
a plurality of multimedia data in relation to time informa- 
tion and/or location information indicating a time and/or 30 
a location at which each multimedia data is created; a 
second computer readable program code for causing 
said computer to obtain the time information and/or the 
location information corresponding to a retrieval request 
upon receiving the retrieval request specified by using 
event names; and a third computer readable program 
code for causing said computer to retrieve multimedia 
data from the plurality of multimedia data managed by 
the first computer readable program code, according to 
the time information and/or the location information ob- 
tained by the second computer readable program code. 
[0033] According to another aspect of the present in- 
vention there is provided an index information providing 
method, comprising: (a) receiving a retrieval request 
specified by using event names, through a network from 
a requestor; (b) obtaining time information and/or loca- 
tion information corresponding to the retrieval request 
received by the step (a) according to the event names 
used in the retrieval request by referring to an index ta- 
ble that contains a plurality of event names and the time 
information and/or the location information that are re- 
lated to each event name, the time information and/or 
the location information indicating a time and/or a loca- 
tion at which each multimedia data is created: and (c) 
providing the time information and/or the location infor- 
mation obtained by the step (b) as an index information, 
through the network to the requestor. 
[0034] According to another aspect of the present in- 



vention there is provided an index server, comprising: a 
request reception unit configured to receive a retrieval 
request specified by using event names, through a net- 
work from a requestor; a processing unit configured to 
obtain time information and/or location information cor- 
responding to the retrieval request received by the re- 
quest reception unit according to the event names used 
in the retrieval request by referring to an index table that 
contains a plurality of event names and the time infor- 
mation and/or the location information that are related 
to each event name, the time information and/or the lo- 
cation information indicating a time and/or a location at 
which each multimedia data is created; and an index in- 
formation transmission unit configured to provide the 
time information and/or the location information ob- 
tained by the processing unit as an index information, 
through the network to the requestor. 
[0035] According to another aspect of the present in- 
vention there Is nrnvirted a computer usable medium 
having computer readable program codes embodied 
therein for causing a computer to function as an index 
server, the computer readable program codes include: 
a first computer readable program code for causing said 
computer to receive a retrieval request specified by us- 
ing event names, through a network from a requestor; 
a second computer readable program code for causing 
said computer to obtain time information and/or location 
information corresponding to the retrieval request re- 
ceived by the first computer readable program code ac- 
cording to the event names used in the retrieval request 
by referring to an index table that contains a plurality of 
event names and the time information and/or the loca- 
tion information that are related to each event name, the 
time information and/or the location information indicat- 
ing a time and/or a location at which each multimedia 
data is created; and a third computer readable program 
code for causing said computer to provide the time in- 
formation and/or the location information obtained by 
the second computer readable program code as an in- 
dex information, through the network to the requestor. 
[0036] According to another aspect of the present in- 
vention there is provided a multimedia data retrieval 
method, comprising: (a) managing each one of a plural- 
ity of multimedia data in relation to time information and/ 
or location Information indicating a time and/or a location 
at which each multimedia data is created; (b) receiving 
a retrieval request specified by using event names, 
through a network from a requestor; (c) obtaining the 
time information and/or the location information corre- 
sponding to the retrieval request received by the step 
(b); (d) retrieving multimedia data from the plurality of 
multimedia data managed by the step (a), according to 
the time information and/or the location information ob- 
tained by the step (c); and (e) providing the multimedia 
data retrieved by the step (d) as a retrieval result, 
through the network to the requestor. 
[0037] According to another aspect of the present in- 
vention there is provided a multimedia data retrieval 
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server, comprising: a data management unit configured 
to manage each one of a plurality of multimedia data in 
relation to time information and/or location information 
indicating a time and/or a location at which each multi- 
media data is created; a request reception unit config- 5 
ured to receive a retrieval request specified by using 
event names, through a network from a requestor; a 
processing unit configured to obtain the time information 
and/or the location information corresponding to the re- 
trieval request received by the request reception unit; a io 
data selection unit configured to retrieve multimedia da- 
ta from the plurality of multimedia data managed by the 
data management unit, according to the time informa- 
tion and/or the location information obtained by the 
processing unit; and a retrieval result transmission unit '5 
configured to provide the multimedia data retrieved by 
the data selection unit as a retrieval result, through the 
network to the requestor. 

[0038] According to another aspect of the present in- 
vention there is provided a computer usable medium 
having computer readable program codes embodied 
therein for causing a computer to function as a multime- 
dia data retrieval server, the computer readable pro- 
gram codes include: a first computer readable program 
code for causing said computer to manage each one of 
a plurality of multimedia data in relation to time informa- 
tion and/or location information indicating a time and/or 
a location at which each multimedia data is created: a 
second computer readable program code for causing 
said computerto receive a retrieval request specified by 
using event names, through a network from a requestor; 
a third computer readable program code for causing 
said computer to obtain the time information and/or the 
location information corresponding to the retrieval re- 
quest received by the second computer readable pro- 
gram code; a fourth computer readable program code 
for causing said computer to retrieve multimedia data 
from the plurality of multimedia data managed by the 
first computer readable program code, according to the 
time information and/or the location information ob- 
tained by the third computer readable program code; 
and a fifth computer readable program code for causing 
said computerto provide the multimedia data retrieved 
by the fourth computer readable program code as a re- 
trieval result, through the network to the requestor. 
[0039] According to another aspect of the present in- 
vention there is provided a method for providing an in- 
dex information providing service from an index server 
to multimedia data retrieval devices which are connect- 
ed through a network, each multimedia data retrieval de- 
vice managing each one of a plurality of multimedia data 
in relation to time information and/or location information 
indicating a time and/or a location at which each multi- 
media data is created, the method comprising: (a) re- 
ceiving a retrieval request specified by using event 
names, at the index server through the network from a 
requesting multimedia data retrieval device; (b) obtain- 
ing the time information and/or the location information 



corresponding to the retrieval request received by the 
step (a) at the index server according to the event 
names used in the retrieval request by referring to an 
index table that contains a plurality of event names and 
the time information and/or the location information that 
are related to each event name; and (c) providing the 
time information and/or the location information ob- 
tained by the step (b) as an index information, from the 
index server through the network to the requesting mul- 
timedia data retrieval device, so as to enable the re- 
questing multimedia data retrieval device to retrieve 
multimedia data from the plurality of multimedia data ac- 
cording to the time information and/or the location infor- 
mation obtained from the index server. 
[0040] According to another aspect of the present in- 
vention there is provided a method for providing a mul- 
timedia data retrieval service from a multimedia data re- 
trieval server to clients which are connected through a 
network, the method comprising: (a) managing each 
one of a plurality of multimedia data at the multimedia 
data retrieval server in relation to time information and/ 
or location information indicating a time and/or a location 
at which each multimedia data is created; (b) receiving 
a retrieval request specified by using event names, at 
the multimedia data retrieval server through a network 
from a requesting client; (c) obtaining the time informa- 
tion and/or the location information corresponding to the 
retrieval request received by the step (b) at the multime- 
dia data retrieval server; (d) retrieving multimedia data 
from the plurality of multimedia data managed by the 
step (a) at the multimedia data retrieval server, accord- 
ing to the time information and/or the location informa- 
tion obtained by the step (c); and (e) providing the mul- 
timedia data retrieved by the step (d) as a retrieval re- 
sult, from the multimedia data retrieval server through 
the network to the requesting client, so as to enable the 
requesting client to obtain the multimedia data matching 
with the retrieval request. 

[0041] Other features and advantages of the present 
invention will become apparent from the following de- 
scription taken in conjunction with the accompanying 
drawings. 

BRIEF DESCRIPTION OF THE DRAWINGS 

[0042] Fig. 1 is a block diagram showing one exem- 
plary configuration of a multimedia data retrieval device 
according to one embodiment of the present invention. 
[0043] Fig. 2 is a diagram showing one exemplary 
manner of data management at a data management unit 
in the multimedia data retrieval device of Fig. 1 . 
[0044] Fig. 3 is a diagram showing another exemplary 
manner of data management at a data management unit 
in the multimedia data retrieval device of Fig. 1 . 
[0045] Fig. 4 is a diagram showing another exemplary 
manner of data management at a data management unit 
in the multimedia data retrieval device of Fig. 1 . 
[0046] Fig. 5 is a diagram showing still another exem- 
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plary manner of data management at a data manage- 
ment unit in the multimedia data retrieval device of Fig. 
1. 

[0047] Fig. 6 is a diagram showing still another exem- 
plary manner of data management at a data manage- 
ment unit in the multimedia data retrieval device of Fig. 
1. 

[0048] Fig. 7 is a diagram showing yet another exem- 
plary manner of data manaaement at a data manage- 
ment unit in the multimedia data retrieval device of Fig. 

1. 

[0049] Fig. 8 is a block diagram showing one exem- 
plary configuration of an index processing unit in the 
multimedia data retrieval device of Fig. 1 . 
[0050] Fig. 9 is a diagram showing an exemplary con- 
tent of an index table used in the index processing unit 
of Fig. 8. 

[0051 ] Fig. 1 0 is a flow chart for an exemplary opera- 
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of Fig. 8. 

[0052] Fig. 11 is a block diagram showing another ex- 
emplary configuration of a multimedia data retrieval de- 
vice according to one embodiment of the present inven- 
tion. 

[0053] Fig. 12 is a block diagram showing an exem- 
plary configuration of an index processing unit in the 
multimedia data retrieval device of Fig. 11. 
[0054] Fig. 13 is a block diagram showing an exem- 
plary configuration of an index server according to one 
embodiment of the present invention. 
[0055] Fig. 14 is a block diagram showing another ex- 
emplary configuration of an index processing unit in the 
multimedia data retrieval device of Fig. 1 . 
[0056] Fig. 15 is a block diagram showing another ex- 
emplary configuration of an index processing unit in the 
multimedia data retrieval device of Fig. 11. 
[0057] Fig. 1 6 is a flow chart for an exemplary opera- 
tion of a data selection unit in the multimedia data re- 
trieval device of Fig. 1 or Fig. 11 . 
[0058] Fig. 17 is a diagram showing one exemplary 
front-end program display screen that can be used in 
the multimedia data retrieval device of Fig. 1 or Fig. 11 . 
[0059] Fig. 18 is a diagram showing an exemplary 
front-end program display screen in the multimedia data 
retrieval device of Fig. 1 or Fig. 11 for one method of 
narrowing down the retrieval result. 
[0060] Fig. 19 is a diagram showing an exemplary 
front-end program display screen in the multimedia data 
retrieval device of Fig. 1 or Fig. 11 for one method of 
narrowing down the retrieval result. 
[0061] Fig. 20 is a diagram showing an exemplary 
front-end program display screen in the multimedia data 
retrieval device of Fig. 1 or Fig. 11 for one method of 
narrowing down the retrieval result. 
[0062] Fig. 21 is a diagram showing an exemplary 
front-end program display screen in the multimedia data 
retrieval device of Fig. 1 or Fig. 11 for another method 
of narrowing down the retrieval result. 



[0063] Fig. 22 is a diagram showing an exemplary 
front-end program display screen in the multimedia data 
retrieval device of Fig. 1 or Fig. 11 for still another meth- 
od of narrowing down the retrieval result. 

5 [0064] Fig. 23 is a diagram showing an exemplary 
front-end program display screen in the multimedia data 
retrieval device of Fig. 1 or Fig. 11 for still another meth- 
od of narrowing down the retrieval result. 
[0065] Fig. 24 is a diagram showing another exempla- 

10 ry front-end program display screen that can be used in 
the multimedia data retrieval device of Fig. 1 or Fig. 11 . 
[0066] Fig. 25 is a diagram showing still another ex- 
emplary front-end program display screen that can be 
used in the multimedia data retrieval device of Fig. 1 or 

15 Fig. 11. 

[0067] Fig. 26 is a schematic diagram showing one 
exemplary configuration of a system using a multimedia 
data retrieval server according to one embodiment of 

thio present invpntinn 

20 [0068] Fig. 27 is a schematic diagram showing anoth- 
er exemplary configuration of a system using a multime- 
dia data retrieval server according to one embodiment 
of the present invention. 

[0069] Rg. 28 is a block diagram showing an exem- 
25 plary configuration of a multimedia data retrieval device 
according to another embodiment of the present inven- 
tion. 

[0070] Fig. 29 is a flow chart for an exemplary opera- 
tion of a reverse look-up retrieval unit in the multimedia 
30 data retrieval device of Fig. 28. 

[0071 ] Fig. 30 is a block diagram showing another ex- 
emplary configuration of a multimedia data retrieval de- 
vice according to another embodiment of the present in- 
vention. 

35 [0072] Fig. 31 is a block diagram showing an exem- 
plary configuration of an index server according to an- 
other embodiment of the present invention. 
[0073] Fig. 32 is a diagram showing one exemplary 
front-end program display screen that can be used in 

40 the multimedia data retrieval device of Fig. 28 or Fig. 30 . 
[0074] Fig. 33 is a diagram showing one conventional 
manner for multimedia data management using directo- 
ries. 

[0075] Fig. 34 is a diagram showing another conven- 
45 tional manner for multimedia data management using 
keywords. 

[0076] Fig. 35 is a diagram showing another conven- 
tional manner for multimedia data management using 
keywords. 

so 

DETAILED DESCRIPTION OF THE PREFERRED 
EMBODIMENTS 

[0077] First, the major features of the present inven- 
55 tion will be briefly summarized. 

[0078] In the present invention, each multimedia data 
is managed by using time information and location in- 
formation that are automatically attached or acquired at 
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or around a time of obtaining the multimedia data, and 
an index table registering a correspondence between 
an event name and the time/location information is pro- 
vided. Then, when a retrieval request specified by using 
the event name is received, the time information and/or 
the location information corresponding to the retrieval 
request are obtained according to the event name, and 
the multimedia data is retrieved according to the ob- 
tained time information and/or location information. 
[0079] In this way, according to the present invention, 
there is no need for tedious tasks of classifying the mul- 
timedia data to be managed into directories or folders 
or attaching keywords, and it becomes possible to real- 
ize the retrieval of the desired multimedia data by using 
an event name or a combination of event names that 
are intuitive and easily comprehensible to users. Also, 
by providing the index table at an index server, it is pos- 
sible to add a new event name at the server side, so that 
it becomes possible to realize the retrieval using a new 
event name without requiring the user to update the in- 
dex table. 

[0080] Referring now to Fig. 1 to Fig. 27, one embod- 
iment of a multimedia data retrieval scheme according 
to the present invention will be described in detail. 
[0081] In a multimedia data retrieval device of the 
present invention, it is presupposed that at least either 
one or both of time information and location information 
are related to the individual multimedia data that is to be 
a retrieval target, and the retrieval, the display, etc., are 
realized by utilizing at least these information. In the fol- 
lowing, this embodiment will be described for an exem- 
plary case of using both the time information and the 
location information. 

[0082] Also, this embodiment is directed to an exem- 
plary case where there is an attribute information (either 
one or both of the time information and the location in- 
formation) that is automatically related to the multimedia 
data by an imaging device or an associated device, and 
this attribute information is utilized. 
[0083] As already mentioned above, in the data for- 
mats for photograph data taken by a digital camera (or 
a digital still camera) and video data taken by a digital 
video camera, the attribute information such as a time 
and a location of the image taking can be recorded. 
There are some digital cameras and digital video cam- 
eras which have a function for automatically recording 
the image taking time and location information as at- 
tributes of data according to these formats, using infor- 
mation obtained from a built-in clock and a GPS (which 
can be built-in in a camera or externally attached to a 
camera) or the like, so that this function can be utilized. 
[0084] There are many digital cameras and digital vid- 
eo cameras which have a function for recording infor- 
mation on the image taking time but no function for re- 
cording a location. In the case of using such a digital 
camera or digital video camera, a portable GPS device 
that can regularly record time and location can be car- 
ried around together with the digital camera or digital 



video camera. The obtained photograph or video has 
the image taking time information attached, so that the 
location information of the image taking time can be de- 
duced from data of the portable GPS device and utilized 

5 as the image taking location information. 

[0085] Besides the GPS, it is also possible to obtain 
the location information by utilizing PHS, Bluetooth, etc. 
In the case where the portable GPS or the like is not 
available, it is also possible to manually attach the loca- 

10 tion information by referring to the map data. 

[0086] With respect to a set of multimedia data to 
which the attribute information such as time and location 
are attached as described above, the multimedia data 
retrieval device of the present invention realizes the re- 

15 trieval based on "events". 

[0087] In the present invention, "events" represent 
various concepts that are related to at least one and at 
most n (n > 1) types of the attribute information among 
the n types of the attribute information that can be given 

20 to the multimedia data, and a name given to such a con- 
cept will be referred to as an "event name". 
[0088] This embodiment is directed to an exemplary 
case of using time and location as the attribute informa- 
tion, so that an event in this embodiment is related to 

25 either one or both of time and location. 

[0089] For example, "Nagano Olympics" is an event 
name, and the period of time and the place at which the 
Nagano Olympics was held are related to this term as 
the time information and the location information re- 

30 spectively. The other event names for which the period 
of time and the place at which each event was held can 
be related as the time information and the location in- 
formation include "Japan EXPO", "Yokohama national 
athletic meeting", "Okinawa Summit" and "Electronics 

35 show '99", for example. 

[0090] The events include those that are held many 
times such as "Gion festival", "Olympics", "Summit" and 
"Electronics show". For such an event, plural sets of the 
time information and the location information indicating 

40 the times and the places at which the event was held 
are related. 

[0091 ] Also, the events include those which have only 
either one of the time information and the location infor- 
mation . For example, the event names to which only the 

45 location information is related include "Shinshu", "Shin- 
juku", "Mt. Fuji". "Yokohama municipal xxx elementary 
school", "xx corporation yy factory", etc. Note however 
that an event name representing a location may also has 
the time information as in the case of a.place name that 

50 was used in the past but is no longer used today or a 
place name that is given only recently. There are also 
event names to which only the time information is relat- 
ed without any fixed location information such as "World 
children year", "Tenth national safety and sanitation 

55 month", etc. There are also events that are held at a 
prescribed period of time every year or once in every 
prescribed number of years such as "New year's day". 
"Christmas" "St. Valentine^ day", "Intercalary year", 
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etc., to which plural sets of the time information are to 
be related. 

[0092] Fig. 1 shows an exemplary configuration of a 
multimedia data retrieval device according to one em- 
bodiment of the present invention. 
[0093] The multimedia data retrieval device 1 of Fig. 
1 generally comprises an index processing unit 1 1 , a da- 
ta selection unit 12 and a data management unit 13. 
rnn94] In the case of using both the time information 
and the location information, the basic operation of each 
constituent element of the multimedia data retrieval de- 
vice 1 of Fig. 1 is as follows. 

[0095] The data management unit 1 3 is managing the 
multimedia data in relation to both the time information 
and the location information. 

[0096] The index processing unit 11 receives an input 
of a retrieval request given by one event name or a com- 
bination of a plurality of event names, and takes out the 



information and the location information corresponding 
to the entered event names (namely, depending on the 
event names, there can be cases where the time infor- 
mation is taken out but there is no location information, 
cases where the location information is taken out but 
there is no time information, and cases where the time 
information and the location information are taken out). 
[0097] Note that, in the case where the input is a re- 
trieval request given by a combination of a plurality of 
event names, it is assumed that a plurality of event 
names will be handled as additive conditions in the ex- 
emplary case of this embodiment, and more specifically, 
this embodiment is directed to an exemplary case of nar- 
rowing down the time information and the location infor- 
mation by carrying out a logical calculation for taking 
AND of a plurality of time information or a plurality of 
location information that are taken out for a plurality of 
event names. It is obviously also possible to specify an 
appropriate logical calculation to be carried out with re- 
spect to a plurality of event names such as AND, OR or 
NOT (in which case it suffices to carry out the specified 
logical calculation such as AND. OR or NOT). 
[0098] The data selection unit 1 2 selects the multime- 
dia data with its related time information/location infor- 
mation matching with the time information or the location 
information, or the time information and the location in- 
formation that are taken out by the index processing unit 
11 in response to the retrieval request, among the mul- 
timedia data managed by the data management unit 13, 
and returns a set of the selected multimedia data as a 
retrieval result. 

[0099] On other other hand, in the case of using either 
one of the time information and the location information, 
the basic operation of each constituent element of the 
multimedia data retrieval device 1 of Fig. 1 is as follows. 
[01 00] The data management unit 1 3 is managing the 
multimedia data in relation to the time information (or 
the location information). 

[01 01 ] The index processing unit 1 1 receives an input 
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of a retrieval request given by one event name or a com- 
bination of a plurality of event names, and takes out the 
time information (or the location information) corre- 
sponding to the entered event names. The case where 
5 the input is a retrieval request given by a combination 
of a plurality of event names will be handled similarly as 
described above. 

[01 02] The data selection unit 1 2 selects the multime- 
dia data with its related time information (or location in- 
fo formation) matching with the time information (or the lo- 
cation information) that is taken out by the index 
processing unit 11 in response to the retrieval request, 
among the multimedia data managed by the data man- 
agement unit 1 3. and returns a set of the selected mul- 
15 timed ia data as a retrieval result. 

[0103] It should be apparent that it is also possible to 
realize the case of using other information in addition to 
the time information, the case of using other information 
in addition to th? location infnrmatinn. or the case of US- 
20 ing other information in addition to the time information 
and the location information. 

[0104] In the following, the case of using both the time 
information and the location information will be mainly 
described. 

25 [0105] Now, as described above, the data manage- 
ment unit 13 of the multimedia data retrieval device of 
this embodiment manages each one of a plurality of re- 
trieval target multimedia data in relation to either one or 
both of the time information and the location information. 

30 [0106] Fig. 2 shows one exemplary manner of realiz- 
ing the multimedia data management at the data man- 
agement unit 13 of the multimedia data retrieval device 
of this embodiment. 

[0107] The example shown in Fig. 2 is directed to a 

35 scheme for utilizing the data formats in which the time 
information and the location information can be includ- 
ed, such as those disclosed in the "digital still camera 
image file format specification (Exif) Version 2.1" (JEI- 
DA-49-1998) by Japan Electronics Industry Develop- 

40 ment Association, and "Recording-Helical-scan digital 
video cassette recording system using 6.35 mm mag- 
netic tape for consumer use (525-80, 625-50, 1125-60 
and 1250-50 systems)" by IEC (International Electro- 
technical Commission). 

45 [0108] In the example shown in Fig. 2, a plurality of 
digital photograph data that are recorded according to 
the data format as defined in the "digital still camera im- 
age file format specification (Exif) Version 2. 1" are man- 
aged under a directory "PHOTO" in the file system, 

so where W P0000100.JAG" to "P0000581 .JPG" are files 
containing the digital photograph data. In each of these 
files : the information on a time at which this photograph 
is taken (denoted as 'TIME:" in Fig. 2) and the informa- 
tion on longitude and latitude as the information on a 

55 location at which this photograph is taken (denoted as 
"LOCATION:" in Fig. 2) are recorded in addition to the 
still image data of the digital photograph. For example, 
the data file "P0000100.JPG" has a value "1970/01/01 
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06:30:05" as the time information and a value 
"E136°47 , 29.37", N34°30'20.05 M " as the location infor- 
mation, which indicate that this is data of a photograph 
taken at 6:30:05 a.m. on January 1, 1970, at 136° 47' 
29.37" of the east longitude and 34° 30' 20.05" D of the 5 
north latitude. 

[01 09] Although Fig. 2 shows an example for the dig- 
ital photograph data, the image taking time and longi- 
tude and latitude of the image taking location for a video 
image can be recorded by using the data format dis- 10 
closed in "Recording-Helical-scan digital video cassette 
recording system using 6.35 mm magnetic tape for con- 
sumer use (525-60, 625-50, 1125-60 and 1250-50 sys- 
tems)" by IEC (International Electrotechnical Commis- 
sion) , for example. Note however that, in contrast to the *5 
still image produced by a digital camera (or a digital still 
camera) or the like where one set of the image taking 
time and the image taking location information is set in 
correspondence to one data, the moving pictures taken 
by a digital video camera may have more than one set 20 
of the image taking time and the image taking location 
information with respect to one data. Namely, there can 
be cases where a plurality of shots taken at different 
times and different locations coexist in one data. 
[0110] It should be apparent to those skilled in the art 25 
that the present invention is equally applicable to digital 
photograph data or digital video data in any data formats 
other than the data formats described above. 
[0111] Fig. 3 and Fig . 4 show another exemplary man- 
ner of realizing the multimedia data management at the 30 
data management unit 13 of the multimedia data retriev- 
al device of this embodiment, where Fig. 4 shows an 
exemplary content of the location information file "LO- 
CATION. DAT* appearing in Fig. 3. 

[01 12] The example shown in Fig. 3 and Fig. 4 is di- 35 
rected to a scheme in which each management target 
data has only the time information, and a correspond- 
ence between each data and its location information is 
separately managed by using a file, a database man- 
agement system, or the like. 

[0113] In the example shown in Fig. 3 and Fig. 4, ten 
digital photograph data from " P0000100.JPG" to 
"P0000581 JPG" are managed under a directory "PHO- 
TO", where each data also records a time it was taken 
as the time information . Here, the data format according 
to Exif Version 2.1 described above can be used as the 
data formal for the digital photograph data. Separately 
from these digital photograph data, a correspondence 
between each photograph and the location information 
indicating a location where it was taken is recorded and 
managed in the location information file. In the example 
shown in Fig. 3 and Fig. 4, the location information file 
has a file name "LOCATION. DAT'. In this "LOCATION. 
DAT, as shown in Fig. 4, a value ,, E136°47 , 29.37". 
N34 o 30'20.05 B " indicating 136° 47' 29.37" of the east 
longitude and 34° 30' 20.05"" of the north latitude is set 
in correspondence to the file name "P00001 00. JPG" as 
the location information of the digital photograph data 



recorded in the file with the file name "P00001 00.JPG", 
for example. 

[0114] Fig. 5 and Fig. 6 show still another exemplary 
manner of realizing the multimedia data management 
at the data management unit 13 of the multimedia data 
retrieval device of this embodiment, where Fig. 6 shows 
an exemplary content of the attribute information file 
"PROPERTY.DAT' appearing in Fig. 5. 
[0115] The example shown in Fig. 5 and Fig. 6 is di- 
rected to a scheme in which each management target 
data has neither the time information nor the location 
information, and a correspondence between each data 
and its time information and location information is sep- 
arately managed by using a file, a database manage- 
ment system, or the like. 

[0116] In the example shown in Fig. 5 and Fig. 6, ten 
digital photograph data from "P0000100.JPG" to 
"P0000581 JPG" are managed under a directory "PHO- 
TO", where each data has neither the time information 
nor the location information. Separately from these dig- 
ital photograph data, a correspondence between each 
photograph and its time information and location infor- 
mation is recorded and managed in the attribute infor- 
mation file. In the example shown in Fig. 5 and Fig. 6, 
the attribute information file has a file name "PROPER- 
TY.DAT*. In this "PROPERTY.DAT, as shown in Fig. 6, 
a value "1970/01/01 06:30:05" indicating 6:30:05 a.m. 
on January 1, 1970 and a value B E136 e 47'29.37", 
N34°30 , 20.05"" indicating 136° 47* 29.37" of the east 
longitude and 34° 30' 20.05"" of the north latitude are 
set in correspondence to the file name "P00001 00. JPG" 
as the time information and location information of the 
digital photograph data recorded in the file with the file 
name "P0000100.JPG". for example. 
[0117] Fig. 7 shows yet another exemplary manner of 
realizing the multimedia data management at the data 
management unit 1 3 of the multimedia data retrieval de- 
vice of this embodiment. 

[0118] The example shown in Fig. 7 is directed to a 
scheme in which each management target data has only 
the time information, and the location information is ob- 
tained by utilizing GPS data. 

[01 1 9] In the example shown in Fig. 7, ten digital pho- 
tograph data from "P00001 00.JAG"to "P0000581 JPG" 
are managed under a directory "PHOTO", where each 
data has the time information but no location informa- 
tion. Separately from these digital photograph data, the 
GPS data obtained at a time of taking each photograph 
is managed. In the example shown in Fig. 7, the GPS 
data for one day is managed as one file, such that a file 
"1 97001 01 .GPS" indicates the GPS data of January 1 , 
1970, a file "1 9700,61 2.GPS" indicates the GPS data of 
June 12, 1970, and so on. In each of these GPS infor- 
mation files, a correspondence table indicating longi- 
tude and latitude corresponding to each timing at a pre- 
scribed time interval is recorded. 
[01 20] By using this GPS information, when the loca- 
tion information of "P0000100.JPG" is necessary, the 
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corresponding GPS information file "1 97001 01 GAS" is 
selected according to the time information of 
"P0000100.JPG" and the location information estimated 
from the time 6:30:05 a.m. is obtained from that infor- 
mation, for example. Here, the GPS information is re- 
corded only at a prescribed time interval (i.e., only dis- 
cretely), so that i I is rare to find the location information 
exactly corresponding to the desired time. For this rea- 
Rnn the location information for a time closest to the 
desired time, or the location information calculated by 
interpolating the location information before and after 
the desired time can be used. 
[0121] Note that various methods described above 
are for the case where there are two types of attribute 
information to be related to the multimedia data, but the 
similar methods can be used for the case where there 
is only one type of attribution information or the case 
where there are three or more types of attribution infor- 
mation. 

[0122] In any of the methods for realizing the multi- 
media data management at the data management unit 
13 described with references to Fig. 2, Figs. 3 and 4, 
Figs. 5 and 6, and Fig. 7, all the photograph data files 
are managed under a single directory called "PHOTO". 
These management methods can be easily used in 
combination with the conventionally known manage- 
ment method for enabling the retrieval using keywords 
or the conventionally known classification method using 
directories. 

[0123] It is also possible to manage the multimedia 
data managed by the data management unit 13 by an- 
other computer by utilizing a mechanism for accessing 
files or data through a network such as a remote file sys- 
tem, instead of the method for managing them within the 
multimedia data retrieval device 1 as described above. 
[0124] Now, the multimedia data retrieval device of 
this embodiment starts its operation upon receiving the 
retrieval request at the index processing unit 11 . The re- 
trieval request can be issued by the user to the multi- 
media data retrieval device 1 directly, by a front-end pro- 
gram that interfaces between the user and the multime- 
dia data retrieval device 1 , or by various application pro- 
grams. 

[0125] The retrieval request is given to the index 
processing unit 11 in a form of a single event name or a 
combination of a plurality of event names. 
[0126] For example, an event name "Shinshu" be- 
comes the retrieval request when it is desired to retrieve 
photograph data taken at Shinsbu, an event name "Na- 
gano Olympics" becomes the retrieval request when it 
is desired to retrieve photograph data taken during a trip 
to see the Nagano Olympics, and an event name "New 
year's day" becomes the retrieval request when it is de- 
sired to retrieve photograph data taken at the new year's 
day. 

[0127] The retrieval condition that cannot be ex- 
pressed by a single event name can be specified by 
combining event names. For example, the retrieval re- 
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quest "New year's day in Kyoto" in which an event name 
"Kyoto" representing a location and an event name 
"New year's day" representing a time are combined can 
be used as the retrieval request for photograph data tak- 
5 en at the new year's day in Kyoto. It is also possible to 
narrow down the retrieval condition further as in the case 
of "New year's day of 1 970 in Kyoto". It should be clearly 
understood that the above description is not intended to 
exclude a combination of basic event names as an event 

10 name, and it is possible to provide a single event name 
such as "New year's day in Kyoto". 

[0128] As other examples, it is also possible to carry 
out the retrieval by specifying only the location informa- 
tion in a form of "event name + its location" or by spec- 
's jfying only the time information in a form of "event name 
+ its time". For example, the event name "Nagano Ol- 
ympics" would be the retrieval request specifying both 
the time information and the location information, but it 
Is also possible to carry nut the retrieval by specifying 
20 only the location information by using "a location of Na- 
gano Olympics", or by specifying only the time informa- 
tion by using "a time of Nagano Olympics". It is also pos- 
sible to narrow down the retrieval condition further by 
using "event name its time + event name" such as "Sap- 
25 poro at a time of Nagano Olympics" 

[0129] Fig. B shows an exemplary configuration of the 
index processing unit 11 in the multimedia data retrieval 
device of this embodiment. 

[0130] As shown in Fig. 8, the index processing unit 
30 11 comprises an index retrieval unit 111 and an index 
table 112. 

[0131] The index retrieval unit 1 1 1 first receives a sin- 
gle event name or a combination of a plurality of event 
names as the retrieval request, and obtains the time in- 

35 formation and the location information corresponding to 
each event name contained in the retrieval request by 
looking up the index table 112. 
[0132] Next, the index retrieval unit 111 gives the ob- 
tained time information and/or location information if the 

40 retrieval condition is a single event name, or the time 
information and/or location information after calculating 
a combination of all the obtained time information and/ 
or location information corresponding to respective 
event names if the retrieval condition is a combination 

45 of a plurality of event names, as the index information 
to the data selection unit 12. 

[0133] The index table 112 is a table for managing a 
correspondence between each event name and n types 
of attribute information its has. Note that there can be 

so cases where one item exists with respect to one type of 
attribute information and cases where a plurality of items 
exist with respect to one type of attribute information (at 
least two items for the start time and the end time will 
be necessary in the case of expressing the time infor- 

55 mation as a period of time, for example). 

[0134] The index table in this embodiment is a table 
for managing either one or both of the correspondence 
between the event name and its time information and 
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the correspondence between the event name and its lo- 
cation information. 

[0135] Fig. 9 shows an example of the index table 112. 
In this example of the index table 112, "start time" and 
"end time" as the time information and "longitude-1", 
"latitude-1", "longitude-2" and "latitude-2" as the loca- 
tion information are managed in correspondence to 
each event name. For example, the event name "Osa- 
ka-jo castle" has no corresponding time information but 
has "longitude-1" and "latitude-1" of a representative 
point as the location information. 
[01 36] In the case of expressing the location informa- 
tion by a representative point, there is an ambiguity re- 
garding an extent of a range centered around that point 
which is intended to be indicated by that event name. 
For this reason, an effective radius distance from the 
representative point may be recorded together in the in- 
dex table. When the radius is not explicitly recorded (or 
in the case of not adopting the method for recording the 
radius explicitly), an effective radius distance from the 
representative point may be set according to appropri- 
ate criteria at the data selection unit 12, for example. 
Note that it is also possible to specify the location infor- 
mation by specifying a representative line rather than a 
representative point. Even in this case, the judgement 
as to how far from that line is intended to be belonging 
to that event can be made similarly as in the case of the 
representative point, by recording that distance together 
in the index table or by setting an effective distance ac- 
cording to appropriate criteria at the data selection unit 
12. 

[0137] Also, in the index table 112 of Fig. 9, the event 
name "Kyoto" has the location information given by the 
longitude-1 • , E135 P 38 , 59.49" , \ the latitude-1 
"NSS^S'lS.ig"", the longitude-2 "EISS'SOW^"". and 
the latitude-2 "N34°54 , 56.38"". Here, the location infor- 
mation of a region indicated by the event name "Kyoto" 
is expressed in approximation by a rectangle with two 
diagonal comers specified by these longitude and lati- 
tude values on a plane. 

[0138] As a method for specifying the location infor- 
mation, besides the method for specifying a rectangular 
region as in the example of Fig. 9, there is also a method 
for using a polygon that approximates the actual region 
more accurately. There is also a method for expressing 
the location information by a set of a plurality of points 
or regions. 

[0139] In the index table 112 of Fig. 9, the event name 
"Japan EXPO" has the start time "1970/3/14" and the 
end time "1970/9/13" as the time information, and the 
longitude-1 M E135°31 "22.20"", the latitude-1 
"N34 ft 48'50.41 "", the longitude-2 "E1 35»32'43.65 M ", and 
the latitude-2 "N34 0 47 , 57.92"" as the location informa- 
tion. Depending on the type of the event or the needs, 
the time information may be specified in further detail by 
specifying the hour as well, or more coarsely by speci- 
fying only the year. 

[0140] The time information or the location informa- 



tion may be directed specified as the event names for 
specifying the retrieval request, in which case there is 
no need to look up the index table. For example, "1 970" 
can be specified to indicate a whole year of 1970 as the 

5 time specification of the retrieval request, or the longi- 
tude and latitude values can be directly specified to in- 
dicate the location specification of the retrieval request. 
This way of specifying the retrieval request can be used 
effectively in the case of combining the event names. 

10 For example, the retrieval request "Kyoto in 1 970" can 
be used to retrieve photographs taken at Kyoto in 1 970. 
[0141] Note that the example of the index table 112 
shown in Fig. 9 records only events related to the digital 
photograph data used in the example shown in Fig. 2, 

'5 but this is only a simplification for the purpose of expla- 
nation, and many more events can be registered in the 
index table in practice. 

[0142] The index retrieval unit 111 of the index 
processing unit 11 in the multimedia data retrieval de- 

20 vice 1 of this embodiment operates according to a pro- 
cedure shown in Fig. 10, for example. 
[0143] The index retrieval unit 111 first receives the 
retrieval request (step S1 ), and next takes out the event 
names contained in that retrieval request (step S2). 

25 Then, the index retrieval unit 1 1 1 looks up the index table 
1 1 2 using the taken out event names as keys, and obtain 
the time information and/or the location information cor- 
responding to each event name (step S3). 
[01 44] Next, the index retrieval unit 1 1 1 calculates the 

30 index information indicating the time information and/or 
the location information of the retrieval target according 
to a way of combining the event names as indicated by 
the retrieval request, based on the time information and/ 
or the location information so obtained (step S4). 

35 [0145] The index retrieval unit 111 then gives the in- 
dex information indicating the time information and/or 
the location information of the retrieval target so ob- 
tained (depending on the event names contained in the 
retrieval request there can be cases where the time in- 

40 formation exists but the location information does not 
exist, cases where the location information exists but the 
time information does not exist, and cases where both 
the time information and the location information exist) 
to the data selection unit 12 (step S5). 

45 [0146] Now, in the configuration of the index process- 
ing unit 11 described above, the index processing unit 
11 locally possesses the index retrieval unit 111 and the 
index table 112 therein. As another exemplary configu- 
ration of the multimedia data retrieval device 1 , it is also 

50 possible to use the configuration as shown in Fig. 11 in 
which the index processing unit 11 carries out the 
processing using a remote index server 3 which is con- 
nected with the multimedia data retrieval device 1 
through a network 6 such as the Internet or an intranet. 

55 [0147] Fig. 12 shows an exemplary configuration of 
the index processing unit 11 in this case. As shown in 
Fig. 12, the index processing unit 11 in this case has a 
retrieval request transmission unit 1 1 3 for sending the 
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received retrieval request to the index server 3, and an 
index information reception unit 11 4 for receiving the in- 
dex information obtained at the index server 3 according 
to this retrieval request and giving it to the data selection 
unit 12. s 
[0148] Fig. 13 shows an exemplary configuration of 
the index server 3 in this case. As shown in Rg. 13, the 
index server 3 in this case has a retrieval request recep- 
tion unit 33 for receiving the retrieval reauest throuah 
the network 6 such as the Internet, an index table 32 for io 
managing event names and theircorrespondingtime in- 
formation and location information, an index retrieval 
unit 31 for obtaining the index information from the re- 
trieval request by referring to the index table 32, and an 
index information transmission unit 34 for returning the 15 
obtained index information. The index retrieval unit 31 
and the index table 32 can be formed to operate similarly 
as those of the index processing unit 11 for carrying out 
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9 and Fig. 10. 20 
[01 49] In the case of using the index server 3, it is also 
possible to use a configuration in which the calculation 
(regarding the time information and the location infor- 
mation) that is to be carried out when the retrieval re- 
quest is specified by a combination of a plurality of event 25 
names will be carried out at the index processing unit 
1 1 side in the multimedia data retrieval device 1 rather 
than at the index server 3 side tin which case, the time 
information and/or the location information correspond- 
ing to each event name can be directly returned from 
the index server 3 to the multimedia data retrieval device 
1 ). This method can be utilized effectively in the case of 
using a plurality of index servers 3 in combination or in 
the case of using the local index table 1 1 2 and the index 
server 3 in combination as will be described below. 
[01 50] The advantage of the index processing unit 1 1 
which has the index table locally is that there is no need 
for the multimedia data retrieval device 1 to have a com- 
munication function. However, there is a need to add 
new events into the index table by some mechanism. 
For example, new events can be delivered by using a 
floppy disk or a CD-ROM, or through the Internet or 
broadcast signals. 

[0151] The advantage of the index processing unit 11 
that uses the index server 3 is that there is no need for 
the index processing unit 11 to carry out the mainte- 
nance of the index table such as addition of new events. 
When the new events are added at the index server 3, 
these new events can be utilized without requiring any 
special operation. Moreover, the index server 3 is con- 
nected to the network 8 so that a plurality of multimedia 
data retrieval devices 1 that are connected to the same 
network 6 can share the single index server 3. 
[0152] As another way of using the index server 3, the 
data of the index table 32 in the index server 3 can be 
cached in the multimedia data retrieval device 1 . Name- 
ly, the data of the event matching with the time informa- 
tion and the location information of the multimedia data 



such as photograph that is managed by the data man- 
agement unit 13 of the multimedia data retrieval device 
1 can be prefetched from the index server 3 and stored 
in the local index table 112 in advance, or the event data 
read out from the index server 3 and used once can be 
cached in the local index table 112. In the subsequent 
retrieval request processing, the index information can 
be obtained by using only the information of the local 
index table 112 whenever it is possible. A range of event 
data to be cached or prefetched can be determined in 
various ways. For example, it is possible to use a variety 
of criteria such as what are to be cached or prefetched 
are all data in the relevant range, only those data that 
are used recently, only those data of a location at which 
many photographs are taken, only those data that are 
looked up frequently, etc. 

[0153] Moreover, It is also possible to provide a plu- 
rality of index servers 3 and switch the index server to 
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formation of a plurality of index servers in combination. 
[0154] For example, by providing a plurality of index 
servers having the same index table, it is possible to dis- 
tribute the processing load among the index servers, or 
it is possible to improve the reliability by making another 
index server available at a time of a trouble in one index 
server. Also, by providing a plurality of index servers 
having different event data, the index server to be used 
can be switched according to the need. Also, by provid- 
ing a plurality of index server having information at dif- 
ferent levels, such as an index server having information 
at the world level, an index server having information at 
Japan level, an index server having information at Kan- 
agawa prefecture level, and so on. for example, it is pos- 
sible to carry out the retrieval by using a plurality of index 
servers ranging from a lower level to a higher level, and 
calculates the index information corresponding to the re- 
trieval request by combining the index information ob- 
tained from these index servers. 
[01 55] Also, the exemplary configurations of the index 
retrieval unit (multimedia data retrieval device) shown 
in Fig. 8 or the index retrieval unit (index server) shown 
in Fig. 1 3, the index retrieval unit looks up only one index 
table, but it is also possible to divide the index table into 
a plurality of tables. 

[0156] Fig. 14 shows an exemplary configuration in 
which the index processing unit 11 of Fig. 8 has a plu- 
rality of index tables. In the example of Fig. 1 4, four index 
tables 112-1 to 112-4 are provided. 
[0157] As a manner of using such a plurality of index 
tables, the events can be classified into a plurality of 
fields and recorded into separate index tables such that 
it becomes easier to manage the index tables. 
[01 58] As another manner of using such a plurality of 
index tables, in Fig. 14, the index table- 1 112-1 can be 
designated as an index table to be shared by a plurality 
of users, the index table-2 1 1 2-2, the index table-3 1 1 2-3 
and the index table-4 1 1 2-4 can be designated as index 
tables for recording events for respective different us- 
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ers, for example. In this way, it becomes possible to use 
the most appropriate index tables according to the us- 
ers, as in the case where one user uses the index table- 
1 112-1 and the index table-2 112-2, another user uses 
the index table-1 112-1 and the index table 112-3, and 
still another user uses the index table-1 112-1 and the 
index table 112-4. In this case, the user specific index 
table can be used to record the personal events, or can 
be used to record events to be shared among a group, 
of users who have the common interest in a particular 
field. 

[0159] More generally, the index retrieval unit 111 can 
be configured to use a necessary number of index tables 
to be shared, index tables for user groups, and personal 
index tables in combination. 

[0160] It should be apparent that the index retrieval 
unit with a plurality of index tables as shown in the ex- 
ample of Fig. 14 can be applied similarly to the index 
retrieval unit 31 of the index server shown in Fig. 13. 
[0161] Next, Fig. 15 shows another exemplary config- 
uration of the multimedia data retrieval device 1 in which 
the index processing unit 11 that has the local index ta- 
ble 112 as shown in Fig. 8 and the remote index server 
3 as shown in Fig. 13 are used in combination. 
[0162] When the retrieval request is received, the in- 
dex processing unit 11 of this example gives the retrieval 
request to both the local index retrieval unit 111 and the 
retrieval request transmission unit 113. 
[0163] The retrieval request transmission unit 113 
sends the retrieval request to the index server 3 through 
the network 6 f and the index information reception unit 
1 14 receives the index information retrieved by the index 
server 3 and gives it to an index information combining 
unit 115. Here, in the case where the retrieval request 
is specified by a combination of a plurality of event 
names, the index server 3 may be configured to retrieve 
the time information and/orthe location information cor- 
respondingto each event name and return them as they 
are or the index server 3 may be configured to calculate 
a combination of the time information and/orthe location 
information for a plurality of events and return the cal- 
culated combination. 

[01 64] Upon receiving the retrieval request the index 
retrieval unit 111 retrieves the time information and/or 
the location information corresponding to each event 
specified in the retrieval request by looking up the local 
index table 1 1 2, and gives them to the index information 
combining unit 115. Here, in the case where the retrieval 
request is specified by a combination of a plurality of 
event names, the index retrieval unit 111 may be con- 
figured to retrieve the time information and/or the loca- 
tion information corresponding to each event name and 
return them as they are to the index information com- 
bining unit 115, or the index retrieval unit 111 may be 
configured to calculate a combination of the time infor- 
mation and/or the location information for a plurality of 
events and return the calculated combination to the in- 
dex information combining unit 115. 



[0165] Note that, in the case where the time informa- 
tion and/orthe location information corresponding to all 
the events specified in the retrieval request cannot be 
obtained by the remote index server 3 or the local index 

5 retrieval unit 1 1 1 alone, the time information and/or the 
location information of those events that can be ob- 
tained by the remote index server 3 or the local index 
retrieval unit 111 alone are obtained and given to the 
index information combining unit 115. 

10 [0166] The index information combining unit 115 cal- 
culates the time information, or the location information, 
or the time information and the location information of 
the retrieval target corresponding to the retrieval request 
by combining the information given from the index re- 

15 trieval unit 1 11 and the index information reception unit 
1 1 4, and gives it to the data selection unit 1 2 as the index 
information. 

[0167] This method for using the local index table in 
the multimedia data retrieval device and the index table 

20 of the index server side in combination is effective in the 
case of carrying out the retrieval using the personal 
event names and the general event names mixedly. 
[0168] For example, in the case of looking for a pho- 
tograph taken at a time of dining at the xx hotel on the 

25 daughter's birthday, the location information corre- 
sponding to the event name "xx hotel" can be retrieved 
by the index server 3. but the personal information such 
as "daughter's birthday" cannot be retrieved by the index 
server 3 which is shared by a plurality of users. For this 

30 reason, the time information of the birthday is registered 
under the event name of "daughter's birthday" in the lo- 
cal personal index table 112 in the multimedia data re- 
trieval device 1 , such that it becomes possible to carry 
out the retrieval by combining this information with the 

35 information obtained by the index server 3. In this way, 
it becomes possible to carry out the retrieval using a 
combination of the general event name registered in the 
index server 3 and the personal event name registered 
in the local index table 1 1 2 such as "daughter's birthday 

40 at the xx hotel", for example (where the location infor- 
mation of the "xx hotel" is obtained from the index server 
3 and the time information of the "daughter's birthday" 
is obtained from the local index table 112). 
[01 69] Note that the method for providing the personal 

45 index table 112 locally in the multimedia data retrieval 
device 1 is described above, but it is also possible to 
provide the personal index table at the index server 3 
side such that the index server 3 carries out the process- 
ing by combining the personal index table and the gen- 

50 eral common index table. 

[0170] It should be apparent that, similarly as the mul- 
timedia data retrieval device 1 having a local index table 
as shown in Fig. 8 can also utilize the remote index serv- 
er in combination, the index server 3 having a local index 

55 table as shown in Fig. 13 can also utilize the other index 
servers in combination. 

[0171] Here, it is also possible to use the personal 
schedule data as the personal index table. For example, 
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the electronic schedule pad managed by a computer 
can record personal schedules in units of day or hour. 
By utilizing this information, the registered schedules 
can be taken as the event names such that, when such 
an event name is specified in the retrieval request, the 5 
date and time of the corresponding schedule are taken 
out as the time information and used in calculating the 
index information. In this way, when a schedule "branch 
manager meetina" is reaistered in the schedule pad and 
the retrieval request "the branch manager meeting held to 
at the industry hair is entered, for example, the index 
information can be calculated by combining the location 
information of the industry hall obtained at the index 
server and the time information of the branch manager 
meeting obtained from the schedule pad. As a method is 
for converting data of the schedule pad into data of the 
index table , the data can be collectively converted at ap- 
propriate timing from the schedule pad to the index ta- 
ble or the schedule psd can be directly referred sit s 
time of processing the retrieval request. 20 
[0172] When the index information is obtained at the 
index processing unit 11 of the multimedia data retrieval 
device of this embodiment the index information is giv- 
en to the data selection unit 12 (see Fig. 1). The data 
selection unit 12 then operates according to a procedure 25 
shown in Fig. 16, for example. 

[0173] Namely, the data selection unit 1 2first receives 
the index information from the index processing unit 1 1 
(step S11). Then, the data selection unit 12 selects 
those multimedia data that match the time information 30 
and/or the location information (there can be cases 
where there is only the time information, cases where 
there is only the location information, and cases where 
there are both the time information and the location in- 
formation) specified by the index information, among the 
multimedia data managed by the data management un it 
13 (step S12), and outputs the selected multimedia data 
as the retrieval result (step S13). 
[01 74] At a time of selecting the multimedia data that 
match the time information and the location information 
specified by the index information (or at a time of select- 
ing the multimedia data that match the location informa- 
tion specified by the index information), there can be 
cases where the location information in the index infor- 
mation only specify the representative point without 
specifying a radius as already mentioned above. The 
data selection unit 12 can be configured to use a pre- 
scribed distance as the radius in such a case or to de- 
termine the radius according to the number of multime- 
dia data that can be displayed on the screen simultane- 
ously in such a case if a front-end program using GUI 
(Graphical User Interface) is used. 
[01 75] It is also possible to configure the data selec- 
tion unit 1 2 such that the retrieval result not only contains 
the multimedia data that match the time information, or 
the location information, or the time information and the 
location information specified by the index information 
as described above, but also those multimedia data 



which do not match exactly but close to the time infor- 
mation or the location information specified by the index 
information. This method is particularly effective in the 
case of carrying out the retrieval through the front-end 
program using GUI. 

[0176] Namely, in the case of displaying the corre- 
sponding photographs on the screen in response to the 
retrieval request, if there are photographs that are not 
corresponding to the retrieval request but are taken at 
nearby locations or at similar times, such photographs 
can be also displayed together such that it becomes 
possible to easily enhance, narrow, or change the re- 
trieval targets by using them as keys. In this case, it is 
preferable to change the manner of display between the 
corresponding photographs and the non-corresponding 
photographs taken at the nearby locations or similar 
times. 

[0177] Note that the processing of the data selection 
unit 1 2c»n hp maris faster by devising the data structure 
such as the use of the B-TREE data structure or the like 
in order to look up the target data from the time informa- 
tion and/or the location information. 
[0178] It is also possible to classify the multimedia da- 
ta managed by the data management unit 13 into 
groups, and specify the target group at a time of select- 
ing the multimedia data at the data selection unit 1 2. For 
example, marks such as "secret photo", "recommended 
photo", "bad photo", etc. can be attached to the multi- 
media data managed by the data management unit 1 3, 
such that the photograph data are retrieved from a set 
of photograph data to which the "recommended photo" 
mark is attached at a time of showing photographs to 
the others, or the photograph data are retrieved also 
from a set of photograph datato which the "secret photo" 
mark is attached at a time of showing photographs to 
intimate friends. 

[0179] In the embodiment described up to this point, 
when the retrieval request is specified by a combination 
of event names, the result of calculating a combination 
of the time information and the location information at 
the index retrieval unit 111 in the index processing unit 
11 is given to the data selection unit 12 as the index in- 
formation. Alternatively, it is also possible to obtain the 
time information and the location information for the in- 
dividual event at the index processing unit 11 and give 
them as they are to the data selection unit 12. In such 
a case, the data selection unit 12 calculates the time 
information and the location information of the retrieval 
target from the time information and the location infor- 
mation for a plurality of events. 
[01 80] Next, the exemplary case of realizing the mul- 
timedia data retrieval device of this embodiment by uti- 
lizing a front-end program using GUI will be described. 
[0181] Fig. 17 shows an exemplary display screen of 
the GUI provided in the multimedia data retrieval device 
of this embodiment. 

[0182] In the exemplary GUI display screen shown in 
Fig. 17. the screen is largely divided into three regions. 
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[0183] A lower left region a1 is a region for entering 
the retrieval request, and Fig. 1 7 shows an exemplary 
case where the retrieval is carried out by specifying an 
event name "New year's day" as the retrieval request. 
[0184] An upper left region a2 is a region for display- s 
ing locations, in which the multimedia data of the retriev- 
al result are displayed at corresponding locations. Here, 
the digital photographs of the retrieval result are dis- 
played in formed of thumb-nail images p1 to p4. In this 
location display region, it is possible to display the re- 10 
trieva! result in superposition to the displayed map data, 
and it is possible to use a more simplified sketch map. 
It is also possible to display the retrieval result by using 
more simple coordinate blocks. It is obviously also pos- 
sible to display the retrieval result alone by accounting is 
forthe relationship among the locations, without display- 
ing anything on the background. 
[0185] A right region a3 of the screen is a region for 
displaying times. The retrieved multimedia data are dis- 
played along a time axis, using the thumb-nail images 20 
or the like. 

[01 86] Note that the GU I display screen can be further 
equipped with various controls such as a retrieval but- 
ton, a scroll bar, check boxes and option buttons accord- 
ing to the needs, but their description will be omitted 25 
here. 

[0187] Now, the example of Fig. 1 7 shows a result of 
the retrieval using the index table of Fig. 9 from the dig- 
ital photograph data shown in Fig. 2, in response to the 
retrieval request given by an event name "New year's 30 
day". From this retrieval result, it can be seen that there 
are photographs taken during a trip to Ise-shima on the 
new yea's day of 1970 and photographs taken during a 
trip to Kyoto on the new year's day of 1 998 in this ex- 
ample. 35 
[0188] Here : this retrieval result can be further nar- 
rowed down to just the photographs of Kyoto, for exam- 
ple. 

[0189] One method for narrowing down the retrieval 
result is to select and specify a portion of Kyoto in the *o 
location display region using a pointer such as a mouse, 
as shown in Fig. 18. In Fig. 18, a dashed line rectangle 
d1 indicates a region selected by a mouse. When such 
a selection is made, the selected portion will be dis- 
played as shown in Fig. 1 9. Here, the time display is also 45 
narrowed down to a portion of the photographs of Kyoto 
in synchronization, but it is also possible to maintain the 
time display unchanged as shown in Fig. 20. At this 
point, the manner of display of the retrieval request field 
can be changed to indicate that the currently displayed so 
retrieval result does not match this retrieval request (it 
is not a display of the retrieval result obtained by using 
this event name), for the convenience of the user, as 
shown in Fig. 19 and fig. 20. 

[0190] Another method for narrowing down the re- 55 
trieval result is to select and specify a portion of 1 998 
on which a trip to Kyoto was made, in the time display 
region using a pointer such as a mouse, as shown in 



Fig. 21 . In Fig. 21 , a dashed line rectangle d2 indicates 
a region selected by a mouse. When such a selection 
is made, the selected portion will be displayed as shown 
in Fig. 1 9. Here, the location display is also narrowed 
down to a portion of the photographs of Kyoto in syn- 
chronization, but it is also possible to maintain the loca- 
tion display unchanged. At this point, the manner of dis- 
play of the retrieval request field can be changed to in- 
dicate that the currently displayed retrieval result does 
not match this retrieval request forthe convenience of 
the user, as shown in Fig. 1 9 and fig. 20. 
[01 91 ] Still another method for narrowing down the re- 
trieval result is to carry out the incremental retrieval by 
adding a new event name "Kyoto" in the retrieval re- 
quest as shown in Fig. 22. When such an incremental 
retrieval is carried out, the selected portion will be dis- 
played as shown in Fig. 23. Here, the location display 
and the time display are also narrowed down to a portion 
of the photographs of Kyoto in synchronization, but it is 
also possible to maintain the location display and the 
time display unchanged. 

[0192] It is also possible to directly specify the time 
information such as "1998" in the retrieval request in- 
stead of the event name such as "New year's day". By 
utilizing this function, it is also possible to realize the 
same narrowing down as shown in Fig. 22 by using a 
combination of "New year's day" and "1998". 
[0193] Note that the multimedia data retrieval device 
that is convenient for the users to carry out the retrieval 
can be realized by combining these methods for narrow- 
ing down the retrieval result with a vicinity retrieval to be 
described below and/or a reverse look-up retrieval to be 
described below. 

[01 94] Now, in the examples of Fig. 1 7 to Fig. 23, the 
thumb-nail images are used for the retrieval result dis- 
played in the location display region and in the time dis- 
play region, so as to make a correspondence between 
these two regions more easily comprehensible. Besides 
that, ft is also possible to display the retrieval result while 
making a correspondence between these two regions 
more easily comprehensible by various methods includ- 
ing a method for using different colors or various marks 
(such as circle, triangle and square marks, or spade, 
heart, diamond and club marks, etc.), a method for dis- 
playing lines connecting between them, and a method 
for displaying a part of one region in highlight in re- 
sponse to an operation made in another region. 
[0195] Fig. 24 shows an exemplary display screen of 
a front-end program that is capable of displaying a pho- 
tograph in an enlarged form on the screen and display- 
ing many photographs on the screen simultaneously. In 
this example, the same retrieval result as in the example 
of Fig. 17 is displayed : but the location display region, 
the time display region and the retrieval request region 
are put in smaller sizes and collected to the left side, 
while a large region for displaying photographs is pro- 
vided on the right side. The correspondences between 
the photographs and their location information and time 
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information are indicated by using marks. Namely, the 
marks appearing below the photographs are displayed 
at the corresponding positions in the location display re- 
gion and the time display region so as to indicate the 
correspondences. 

[0196] Next, Fig. 25 shows an exemplary display 
screen showing a retrieval result obtained by the retriev- 
al using an event name "Japan EXPO" as the retrieval 
rpniipst In this example, in addition to the Dhotoaraohs 
taken at the Japan EXPO in 1970, a photograph of the 
"Osaka-jo castle" that was taken on the way back from 
that trip is also displayed. This vicinity retrieval function 
for displaying data of close locations or similar times to- 
gether with the data corresponding to the retrieval re- 
quest is very effective in the case where the user carries 
out the retrieval based on vague recollections. 
[0197] Note that, in the case of simultaneously dis- 
playing the retrieval result that corresponds to the re- 
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not corresponds to the retrieval target exactly, it is also 
passible to indicate the difference between them by us- 
ing different display manners for them. For example, the 
vicinity retrieval result can be displayed in a lightertone, 
or in a different color, or in a smaller size, or by using a 
different mark, etc. 

[0198] It is also possible to enable the user to select 
whether or not to utilize the vicinity retrieval function in 
the case of providing this function in the multimedia data 
retrieval device. 

[0199] Note that the multimedia data retrieval device 
for carrying out the retrieval through the front-end pro- 
gram using GUI as described above (which can be re- 
alized in a form of utilizing the index server or in a form 
of not utilizing the index server, which can be realized 
in a form of managing the multimedia data by this device 
itself or in a form of managing the multimedia data by 
another computer, and which can be realized in many 
other variations as already described above) can be re- 
alized by displaying the GUI display screen on a com- 
puter executing the front-end program and a program 
(which will be referred to as a multimedia data retrieval 
program) corresponding to functions of the multimedia 
data retrieval device, but it is also possible to realize this 
multimedia data retrieval device as follows. 
[0200] For example, as shown in Fig. 26, in the case 
of the multimedia data retrieval device not utilizing the 
index server, a computer 301 (which will be referred to 
as a client) for displaying the GUI display screen and a 
computer (which will be referred to as a multimedia data 
retrieval server) 302 for executing a front-end program 
321 and a multimedia data retrieval program 322 can 
be provided as separate devices that are connectable 
through the network 6 such as the I ntemet or an intranet, 
such that the front-end program 321 on the multimedia 
data retrieval server 302 can be operated through a 
WEB browser 311 on the client 301 . 
[0201] In this case, the front-end program and the 
multimedia data retrieval program can be operated from 



any computer (a computer on which the WEB browser 
is mounted) on the network. 

[0202] The similar configuration is also applicable to 
the case of the multimedia data retrieval device utilizing 

5 one or a plurality of index servers 3 as shown in Fig. 27. 
[0203] Note that it is also possible to manage the mul- 
timedia data by another computer (which will be referred 
to as a multimedia data management server) connected 
to the network 6 in the configuration of Fig. 26 or Fig. 27. 

10 [0204] Referring now to Fig. 28 to Fig. 32, another em- 
bodiment of a multimedia data retrieval scheme accord- 
ing to the present invention will be described in detail. 
[0205] The multimedia data retrieval device of this 
embodiment has a reverse look-up retrieval function for 

15 retrieving corresponding one or a plurality of event 
names by specifying a specific multimedia data, in ad- 
dition to the functions of the multimedia data retrieval 
device of the previous embodiment. By using this re- 
verse !ook-'jp retrieval function, 't hpnomfis possible to 

20 obtain information useful in ascertaining what is imaged 
on a specific photograph. Also, by carrying out the re- 
trieval again by using the event names so obtained, it 
also becomes possible to retrieve photographs that are 
related to some photograph. 

25 [0206] In the following, this new function to be added 
to the previous embodiment will be mainly described. 
[0207] The reserve look-up retrieval function can be 
realized by looking up the index table for looking up the 
time information and the location information regarding 

30 the event from the event name, in a reverse direction. 
Namely, the index table is usually looked up by using 
the event name as a key, but in the case of the reverse 
look-up retrieval, events that contain either one or both 
of the specified time information and location inform a- 

35 tion (or events that are not corresponding to them ex- 
actly but corresponding to close locations or similar 
times in addition to them) are retrieved. 
[0208] Fig. 28 shows an exemplary configuration of 
the multimedia data retrieval device which has this re- 

40 verse look-up retrieval function. Fig. 28 shows a config- 
uration in which the reverse look-up retrieval function is 
provided in the multimedia data retrieval device having 
a local index table as shown in Fig. 1 and Fig. 8. 
[0209] A reverse look-up retrieval request of the user 

45 is entered into a reverse look-up retrieval unit 14, and 
the reverse look-up retrieval unit 1 4 carries out the re- 
verse look-up retrieval by utilizing the multimedia data 
managed by the data management unit 1 3 and the index 
table 111 of the index processing unit 12 and outputs 

50 the retrieval result. 

[0210] Fig. 29 shows an exemplary procedure of the 
operation of the reverse look-up retrieval unit 14. 
[0211] The reverse look-up retrieval unit 14 first re- 
ceives the reverse look-up retrieval request from the us- 

55 er (step S21 ). The user requests the reverse look-up re- 
trieval by specifying the multimedia data such as a dig- 
ital photograph from which the reverse look-up of the 
event name is desired. At this point, as a method for 
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specifying the multimedia data, a name that can unique- 
ly identify the multimedia data managed by the data 
management unit 13 can be used, In the case of spec- 
ifying the reverse look-up retrieval through the front-end 
program using GUI, it is possible to issue the reverse 
look-up retrieval request using a name for uniquely iden- 
tifying a photograph displayed on the screen when that 
photograph is specified on the screen, for the conven- 
ience of the user. 

[0212] Upon receiving the reverse look-up retrieval 
request, the reverse look-up retrieval unit 14 takes out 
the retrieval target specification, such as an identifier for 
specifying the multimedia data to be retrieved by the re- 
verse look-up retrieval for example, from that reverse 
look-up retrieval request (step S22). 
[0213] Next, the reverse look-up retrieval unit 14 
takes out the time Information and the location informa- 
tion of the retrieval target multimedia data that is man- 
aged by the data management unit 1 3 (step S23). Then, 
the reverse look-up retrieval unit 14 retrieves those 
events which have the time information and the location 
information that contain the time information and the lo- 
cation information of the retrieval target so obtained, 
from the index table 112 (step S24). 
[021 4] Note that there are various methods for realiz- 
ing this reverse look-up retrieval, including a method for 
retrieving events corresponding to both the time infor- 
mation and the location information of the specified mul- 
timedia data, a method for retrieving events which have 
only one of the time information and the location infor- 
mation which matches the specified time information or 
location information in addition, a method for retrieving 
events which have at least one of the time information 
and the location information that matches the specified 
time information and location information, and a method 
for retrieving events which have at least one of the time 
information and the location information that does not 
match the specified one exactly but that has a value 
closer to the specified one in addition to any of the 
events described above. 

[0215] Note also that the number of events that can 
be retrieved here is not necessarily one and a plurality 
of events may be retrieved. 

[0216] One or a plurality of event names retrieved in 
this way are then returned as a reverse look-up retrieval 
result (step S25). In the case where the reverse look-up 
retrieval result contains a plurality of event names, the 
event that includes the retrieval target or the event that 
is closer to the retrieval target in terms of time informa- 
tion or the location information can be given a higher 
priority level, and the retrieved event names can be re- 
arranged in a descending order of the priority levels, so 
as to enable more effective utilization of the retrieval re- 
sult. It is also possible to return the retrieval result by 
attaching a score indicating the priority level to each 
event name. 

[021 7] It should be apparent that this reverse look-up 
retrieval function is also applicable to the multimedia da- 



ta retrieval device which utilizes the index server as 
shown in fig. 11. 

[0218] Fig. 30 shows an exemplary configuration of 
the multimedia data retrieval device which has the re- 

5 verse look-up retrieval function utilizing the index server. 
[0219] In this exemplary configuration, the reverse 
look-up retrieval unit 14 comprises a reverse look-up re- 
trieval request transmission unit 143 and a reverse look- 
up retrieval result reception unit 144. 

10 [0220] The reverse look-up retrieval request from the 
user is entered into the reverse look-up retrieval request 
transmission unit 143. The reverse look-up retrieval re- 
quest transmission unit 143 first takes out the retrieval 
target multimedia data specified in the reverse look-up 

15 retrieval request of the user, and obtains the time infor- 
mation and/or the location information of the specified 
retrieval target multimedia data from the data manage- 
ment unit 13. The time information and/or the location 
information of the retrieval target so obtained are then 

20 transmitted through the network 6 as the reverse look- 
up retrieval request to the index server 3 having the re- 
verse look-up retrieval function. The index server 3 
transmits the reverse look-up retrieval result through the 
network 6. The reverse look-up retrieval result reception 

25 unit 144 receives the reverse look-up retrieval result 
from the index server 3 and outputs it as the reverse 
look-up retrieval result. 

[0221] Fig. 31 shows an exemplary configuration of 
the index server 3 having the reverse look-up retrieval 
30 function. 

[0222] In this exemplary configuration, the retrieval 
request sent to the index server 3 is received by a re- 
trieval request reception unit 33, and given to an index 
retrieval unit 31 if it is an ordinary retrieval request or to 
35 a reverse look-up retrieval unit 34 if it is a reverse look- 
up retrieval unit 34. 

[0223] The operation of the index retrieval unit 31 is 
the same as that in the index server of Fig. 13. 
[0224] The reverse look-up retrieval unit 34 retrieves 

40 those events which have the time information and the 
location information that contain the time information 
and the location information of the retrieval target spec- 
ified in the reverse look-up retrieval request, or those 
events which have the time information and the location 

45 information close to the time information and the loca- 
tion information of the retrieval target, from the index ta- 
ble 32 as described above, by looking up the index table 
32 that is shared with the index retrieval unit 31 . Then, 
the retrieved one or a plurality of event names are given 

50 to the retrieval result transmission unit 35 as the retrieval 
result. 

[0225] The retrieval result transmission unit 35 then 
returns the retrieval result given from the index retrieval 
unit 31 or the reverse look-up retrieval unit 34 to the mul- 
55 timedia data retrieval device 1 that issued the retrieval 
request. 

[0226] Fig. 32 shows an exemplary GUI display 
screen in the multimedia data retrieval device having the 
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reverse look-up retrieval function which is configured to 
be utilized through a front-end program using GUI. The 
example of Fig. 32 is a display screen showing a result 
of the retrieval of digital photograph data by using an 
event name "New year's day" as the retrieval request, 
similarly as in the example of Fig. 24. Here, unlike the 
example of Fig. 24, an event name of an event that is 
supposed to be shown in each photograph is displayed 
Mr\\u oarh nhntnnranh in the photograph display region 
on the right side. This event name is obtained by carry- 
ing out the reverse look-up retrieval with respect to each 
photograph data and selecting one event name with the 
highest priority level among the retrieved event names. 
[0227] There are various methods that can be used 
as a method for displaying the reverse look-up retrieval 
result. For example, when a plurality of event names are 
retrieved with respect to one photograph, a prescribed 
number of event names are displayed in the descending 
ui utM of the pnonty levels, or only one event nemo with 
the highest priority level is displayed along with an indi- 
cation that there are other retrieved event names such 
that the other event names can be viewed by an oper- 
ation such as a clicking of an icon, for example. It is also 
possible to display the event names retrieved by the re- 
verse look-up retrieval at corresponding locations on the 
map in the location display region. 
[0228] It should be apparent to those skilled in the art 
that the multimedia data retrieval device having the re- 
verse look-up retrieval function of this embodiment can 
be modified similarly as the multimedia data retrieval de- 
vice of the previous embodiment, by providing a plurality 
of index tables, or by utilizing the local index table and 
the remote index server in combination. 
[0229] Also, in the case of carrying out the retrieval 
through a front-end program using GUI, a computer for 
displaying the GUI screen can be provided separately 
such that the front-end program and the multimedia data 
retrieval program can be operated from any computer 
with the WEB browser mounted thereon on the network, 
similarly as in the previous embodiment. Besides these, 
the various exemplary configurations and variations de- 
scribed for the multimedia data retrieval device of the 
previous embodiment are also applicable to the multi- 
media data retrieval device ot this embodiment. 
[0230] Note that the retrieval target data of the multi- 
media data retrieval device of the present invention are 
not limited to still images of digital cameras or scanners 
or moving pictures of digital video cameras or digital 
broadcasting as in the embodiments described above, 
and can be any data as long as data are managed in 
relation to either one or both of the time information and 
the location information. For example, with respect to 
various events such as sightseeing sites, festivals, con- 
certs, exhibitions, athletic competitions, etc., it is also 
possible to similarly manage and retrieve any of docu- 
ments describing these events, links to the WEB pages 
containing Information regarding these events, links to 
encyclopedia sections that explain these events, links 



to on-line shops that sell tickets or records of these 
events, etc. 

[0231 ] Note also that it is preferable for the multimedia 
data retrieval device of the present invention to handle 
5 the time information of each event or the time informa- 
tion that is related to the multimedia data by accounting 
for the time differences. 

[0232] Note also that it is preferable for the multimedia 
data retrieval device of the present invention to register 
10 each event name in a plurality of languages (such as 
Japanese, English, French, Spanish, Italian, Chinese, 
Russian, etc.) such that the retrieval can be carried out 
in different languages. 

[0233] As described, according to the present inven- 
ts tion, the multimedia data are managed by using the time 
information and the location information, and when a re- 
trieval request specified by using one or a plurality of 
event names is received, the multimedia data are re- 

triouort K»# ncinn iha timo infnrmptinn anri/nrthfi location 

20 information corresponding to the retrieval request that 
are obtained according to the event names from the in- 
dex table or the index server, so that there is no need 
for tedious tasks of classifying the multimedia data to be 
managed into directories or folders or attaching key- 

25 words for the purpose of the retrieval, and it becomes 
possible to retrieve the desired multimedia data by using 
an event name or a combination of event names that 
are intuitive and easily comprehensible for the users. 
[0234] it is to be noted that the above described em- 

30 bodiments according to the present invention may be 
conveniently implemented using a conventional general 
purpose digital computer programmed according to the 
teachings of the present specification, as will be appar- 
ent to those skilled in the computer art. Appropriate soft- 

35 ware coding can readily be prepared by skilled program- 
mers based on the teachings of the present disclosure, 
as will be apparent to those skilled in the software art. 
[0235] In particular, the multimedia data retrieval de- 
vice or the index server of each of the above described 

40 embodiments can be conveniently implemented in a 
form of a software package. 

[0236] Such a software package can be a computer 
program product which employs a storage medium in- 
cluding stored computer code which is used to program 

45 a computer to perform the disclosed function and proc- 
ess of the present invention. The storage medium may 
include, but is not limited to, any type of conventional 
floppy disks, optical disks, CD-ROMs, magneto-optical 
disks, ROMs, RAMs, EPROMs, EEPROMs, magnetic 

so or optical cards, or any other suitable media for storing 
electronic instructions. 

[0237] It is also to be noted that, besides those al- 
ready mentioned above, many modifications and varia- 
tions of the above embodiments may be made without 
55 departing from the novel and advantageous features of 
the present invention. Accordingly, all such modifica- 
tions and variations are intended to be included within 
the scope of the appended claims. 
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Claims 



7. A multimedia data retrieval device, comprising: 



1 . A multimedia data retrieval method, comprising: 

(a) managing each one of a plurality of multi- 
media data in relation to time information and/ 
or location information indicating a time and/or 
a location at which each multimedia data is cre- 
ated; 

(b) obtaining the time information and/or the lo- 
cation information corresponding to a retrieval 
request upon receiving the retrieval request 
specified by using event names; and 

(c) retrieving multimedia data from the plurality 
of multimedia data managed by the step (a) . ac- 
cording to the time information and/or the loca- 
tion information obtained by the step (b). 

2. The method of claim 1 , wherein the step (b) obtains 
the time information and/or the location information 
corresponding to the retrieval request by referring 
to an index table that contains a plurality of event 
names and the time information and/or the location 
information that are related to each event name. 

3. The method of claim 2, wherein the step (b) utilizes 
schedule data as the index table, by taking name 
information indicating schedule contents as the 
event names, date and time information of sched- 
ules as the time information, and place information 
of schedules as the location information. 

4. The method of claim 1 , wherein the step (b) obtains 
the time information and/or the location information 
corresponding to the retrieval request from an index 
server by transmitting the retrieval request to the in- 
dex server which has a function for obtaining the 
time information and/or the location information that 
are related to each event name. 

5. The method of claim 1 , further comprising: 

(d) obtaining the time information and/or the lo- 
cation information that are related to one mul- 
timedia data upon receiving a reverse look-up 
retrieval request specifying said one multime- 
dia data; and 

(e) retrieving event names according to the time 
information and/or the location information ob- 
tained by the step (d) as a reverse look-up re- 
trieval result. 

6. The method of claim 1 , wherein the step (a) man- 
ages each multimedia data in relation to the time 
information and/or the location information which 
are automatically created in relation to a creation of 
each multimedia data. 



a data management unit configured to manage 
each one of a plurality of multimedia data in re- 

5 lation to time information and/or location infor- 

mation indicating a time and/or a location at 
which each multimedia data is created; 
a processing unit configured to obtain the time 
information and/or the location information cor- 

10 responding to a retrieval request upon receiv- 

ing the retrieval request specified by using 
event names; and 

a data selection unit configured to retrieve mul- 
timedia data from the plurality of multimedia da- 
is ta managed by the data management unit, ac- 

cording to the time information and/or the loca- 
tion information obtained by the processing 
unit. 

20 8. A computer usable medium having computer read- 
able program codes embodied therein for causing 
a computer to function as a multimedia data retriev- 
al device, the computer readable program codes in- 
clude: 

25 

a first computer readable program code for 
causing said computer to manage each one of 
a plurality of multimedia data in relation to time 
information and/or location information indicat- 

30 ing a time and/or a location at which each mul- 

timedia data is created; 
a second computer readable program code for 
causing said computer to obtain the time infor- 
mation and/or the location information corre- 

35 sponding to a retrieval request upon receiving 

the retrieval request specified by using event 
names; and 

a third computer readable program code for 
causing said computer to retrieve multimedia 
^0 data from the plurality of multimedia data man- 

aged by the first computer readable program 
code, according to the time information and/or 
the location information obtained by the second 
computer readable program code. 

45 

9. An index information providing method, comprising: 

(a) receiving a retrieval request specified by us- 
ing event names, through a network from a re- 

50 questor; 

(b) obtaining time information and/or location 
information corresponding to the retrieval re- 
quest received by the step (a) according to the 
event names used in the retrieval request by 

55 referring to an index table that contains a plu- 

rality of event names and the time information 
and/or the location information that are related 
to each event name, the time information and/ 



19 



BNSDOCID: <EP 1139239A2J_> 



37 



EP1 139 239 A2 



38 



or the location information indicating a time 
and/or a location at which each multimedia data 
is created; and 

(c) providing the time information and/or the lo- 
cation information obtained by the step (b) as 
an index information, through the networktothe 
requestor. 

10. The method of claim 9. wherein the index table in- 
cludes a plurality of personal index tables and a 
common index table, and the step (b) obtains the 
time information and/or the location information by 
referring at least to one of the personal index table 
corresponding to the requestor and the common in- 
dex table. 

11. The method of claim 9, wherein the step (b) utilizes 
schedule data as the index table, by taking name 



event names, date and time information of sched- 
ules as the time information, and place information 
of schedules as the location information. 

12. An index server, comprising: 

a request reception unit configured to receive 
a retrieval request specified by using event 
names, through a network from a requestor; 
a processing unit configured to obtain time in- 
formation and/or location information corre- 
sponding to the retrieval request received by 
the request reception unit according to the 
event names used in the retrieval request by 
referring to an index table that contains a plu- 
rality of event names and the time information 
and/or the location information that are related 
to each event name, the time information and/ 
or the location information indicating a time 
and/or a location at which each multimedia data 
is created; and 

an index information transmission unit config- 
ured to provide the time information and/or the 
location information obtained by the processing 
unit as an index Information, through the net- 
work to the requestor. 

13. A computer usable medium having computer read- 
able program codes embodied therein for causing 
a computer to function as an index server, the com- 
puter readable program codes include: 

a first computer readable program code for 
causing said computer to receive a retrieval re- 
quest specified by using event names, through 
a network from a requestor; 
a second computer readable program code for 
causing said computer to obtain time informa- 
tion and/or location information corresponding 



to the retrieval request received by the first 
computer readable program code according to 
the event names used in the retrieval request 
by referring to an index table that contains a 
plurality of event names and the time informa- 
tion and/or the location information that are re- 
lated to each event name, the time information 
and/or the location information indicating a time 
and/or a location at which each multimedia data 
is created; and 

a third computer readable program code for 
causing said computer to provide the time in- 
formation and/or the location information ob- 
tained by the second computer readable pro- 
gram code as an index information, through the 
network to the requestor. 

14. A multimedia data retrieval method, comprising: 

(a) managing each one of a plurality of multi- 
media data in relation to time information and/ 
or location information indicating a time and/or 
a location at which each multimedia data is cre- 
ated; 

(b) receiving a retrieval request specified by us- 
ing event names, through a network from a re- 
questor; 

(c) obtaining the time information and/or the lo- 
cation information corresponding to the retriev- 
al request received by the step (b); 

(d) retrieving multimedia data from the plurality 
of multimedia data managed by the step (a), ac- 
cording to the time information and/or the loca- 
tion information obtained by the step (c); and 

(e) providing the multimedia data retrieved by 
the step (d) as a retrieval result, through the 
network to the requestor. 

15. The method of claim 14, wherein the step (c) ob- 
tains the time information and/or the location infor- 
mation corresponding to the retrieval request by re- 
ferring to an index table that contains a plurality of 
event names and the time information and/or the 
location information that are related. to each event 
name. 

16. The method of claim 14, further comprising: 

(f) obtaining the time information and/or the lo- 
cation information that are related to one mul- 
timedia data upon receiving a reverse look-up 
retrieval request specifying said one multime- 
dia data; and 

(g) retrieving event names according to the time 
information and/or the location information ob- 
tained by the step (f) as a reverse look-up re- 
trieval result. 
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17. A multimedia data retrieval server, comprising: 

a data management unit configured to manage 
each one of a plurality of multimedia data in re- 
lation to time information and/or location infor- 5 
mation indicating a time and/or a location at 
which each multimedia data is created; 
a request reception unit configured to receive 
a retrieval request specified by using event 
names, through a network from a requestor; 10 
a processing unit configured to obtain the time 
information and/or the location information cor- 
responding to the retrieval request received by 
the request reception unit; 
a data selection unit configured to retrieve mul- 15 
timedia data from the plurality of multimedia da- 
ta managed by the data management unit, ac- 
cording to the time information and/or the loca- 
tion information obtained by the processing 
unit; and 20 
a retrieval result transmission unit configured 
to provide the multimedia data retrieved by the 
data selection unit as a retrieval result, through 
the network to the requestor. 

25 

18. A computer usable medium having computer read- 
able program codes embodied therein for causing 
a computer to function as a multimedia data retriev- 
al server, the computer readable program codes in- 
clude: 30 

a first computer readable program code for 
causing said computer to manage each one of 
a plurality of multimedia data in relation to time 
information and/or location information indicat- 35 
ing a time and/or a location at which each mul- 
timedia data is created; 
a second computer readable program code for 
causing said computer to receive a retrieval re- 
quest specified by using event names, through 40 
a network from a requestor; 
a third computer readable program code for 
causing said computer to obtain the time infor- 
mation and/or the location information corre- 
sponding to the retrieval request received by 45 
the second computer readable program code; 
a fourth computer readable program code for 
causing said computer to retrieve multimedia 
data from the plurality of multimedia data man- 
aged by the first computer readable program so 
code, according to the time information and/or 
the location information obtained by the third 
computer readable program code; and 
a fifth computer readable program code for 
causing said computer to provide the multime- ss 
dia data retrieved by the fourth computer read- 
able program code as a retrieval result, through 
the network to the requestor. 



19. A method for providing an index information provid- 
ing service from an index server to multimedia data 
retrieval devices which are connected through a 
network, each multimedia data retrieval device 
managing each one of a plurality of multimedia data 
in relation to time information and/or location infor- 
mation indicating a time and/or a location at which 
each multimedia data is created, the method com- 
prising: 

(a) receiving a retrieval request specified by us- 
ing event names, at the index server through 
the network from a requesting multimedia data 
retrieval device; 

(b) obtaining the time information and/or the lo- 
cation information corresponding to the retriev- 
al request received by the step (a) at the index 
server according to the event names used in 
the retrieval request by referring to an index ta- 
ble that contains a plurality of event names and 
the time information and/or the location infor- 
mation that are related to each event name; and 

(c) providing the time information and/or the lo- 
cation information obtained by the step (b) as 
an index information, from the index server 
through the network to the requesting multime- 
dia data retrieval device, so as to enable the 
requesting multimedia data retrieval device to 
retrieve multimedia data from the plurality of 
multimedia data according to the time informa- 
tion and/or the location information obtained 
from the index server. 

20. A method for providing a multimedia data retrieval 
service from a multimedia data retrieval server to 
clients which are connected through a network, the 
method comprising: 

(a) managing each one of a plurality of multi- 
media data at the multimedia data retrieval 
server in relation to time information and/or lo- 
cation information indicating a time and/or a lo- 
cation at which each multimedia data is creat- 
ed; 

(b) receiving a retrieval request specified by us- 
ing event names, at the multimedia data retriev- 
al server through a network from a requesting 
client; 

(c) obtaining the time information and/or the lo- 
cation information corresponding to the retriev- 
al request received by the step (b) at the multi- 
media data retrieval server; 

(d) retrieving multimedia data from the plurality 
of multimedia data managed by the step (a) at 
the multimedia data retrieval server according 
to the time information and/or the location infor- 
mation obtained by the step (c); and 

(e) providing the multimedia data retrieved by 
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the step (d) as a retrieval result, from the mul- 
timedia data retrieval server through the net- 
work to the requesting client, so as to enable 
the requesting client to obtain the multimedia 
data matching with the retrieval request. 5 
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